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Human-electronic  Crewmember 
Conferences 


THE  HIWAN-ELECTRONIC  CREV:  CAN  THEY  WORK  TOGETHER  ? 
Conference  Roon  at  Stadttheater  Ingolstadt 
Ingolstadt,  FRG,  19  -  22  Sept  1988 


THE  HUMAN -ELECTRONIC  CREW: 
IS  THE  TEAM  MATURING? 

The  2nd  Joint  OAF/ RAF/US AF  Workshop 
on  Human  -  Electronic  Crew  Teamwork 

St*dtth«t«r  C«*f«r*nc*  Rm«,  FK 

25-21  Wt**6*r  IWO 


THE  HUMAN-ELECTRONIC  CREW: 
CAN  WE  TRUST  THE  TEAM? 

Proceeding*  of  the  3rd  International  Workshop  on 
Human-Computer  Teamwork 
Cambridge.  United  Kingdom.  27-30  September  1994 


THE  HUMAN-ELECTRONIC  CREW: 

THE  RIGHT  STUFF? 
Proceedings  of  the  4th  Joint  GAF/RAF/USAF 
Workshop  on  Human-Computer  Teamwork 
Hotel  zur  Post,  Kreuth,  Nr  Tegemsoe.  Germany 
23  -  26  September.  1997 


THREE  PHASES  OF  SUPPORT 
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Supervisory  Control 


“  Specifically,  supervisory  control  means 
that  human  operators  plan  and  teach 
tasks  to  computers,  and  the  computers 
then  implement  the  tasks  automatically 
through  their  own  sensory  inputs,  memory 
and  control  actions” 


Sheridan,  p.450,  in  Par«*im*n  &  Motic-uj,  1006. 


v  Personal  Satellite  Assistant  Application 
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Predator  and  Ground  Control  Stations 


Predator  Ground  Control  Station 
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Tactical  Control  Station 

hrtp/ftvvTM  rmr  nnvy  Mmi 
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UAV  Ground  Control  Station 
Simulator 
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Camera  Display 
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Head  Down 
Displays 


r  Vehicle  Operator  (AVO) 
^Ratcd  pilot 
•Controls  aircrall/systcm 

■ 

•Flight  commander 


Sensor  Operator  (SO) 


'Imagery'  analyst 
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Tactile  Displays] 
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Haptic  Feedback 
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Interface  Technologies 


"  Synthetic  Overlays 
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Speech  Interfaces 


Symbology  Improvements 
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Rationale  for  Speech  Input  in 
UAV  Control  Stations 


•  Natural  and  intuitive  interface 

•  Mature  technology 

•  Real-time 

•  Accurate 

■  Speaker-independent 

•  Reduces  resource  competition 

•  Hands-free  control 

•  Heads-up  control 

•  “Voice  macros”  can  replace 
complex  series  of  manual  inputs 


Speech  input  improved 
performance  in  manned 
aircrew  applications 
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Speech  Input  Implementation 


•  Speech  Recognizer: 

•Nuance  v8.0 

•continuous,  speaker-independent 

•  Vocabulary: 

•70  commands 


NUANCE 


IhtfkfWttdfeBBWI 


•Total  vocabulary:  160 

•  Push-to-Talk  button  (on  joystick) 

•  Sennheiser  280-13  Pro  headset/mic 


•  Visual  feedback  of  commands  on  HUD 


•  Auditory  environment:  benign 
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Results  -  Speech  Better: 
Mean  Task  Completion  Time 


Level  Emerg  Data  Trans  Speed  Servo  Icing  Info 
Off  Waypt  OverHt  OH  Differ  OH _ Query 

Speech  input  significantly  faster  for  all  task  types 
Average  time  savings  range:  3.14  to  21. 43  seconds 
Data  entry  time  reduced  by  approx  40%  with  Speech  Input 


Results  -  Speech  Better: 
Mean  Task  Completion  Accuracy 


Speech  Input  more  Accurate  than  Manual 
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•  More  tasks  completed  correctly  with  Speech: 

Voice  macros  are  a  factor 

•  Time  outs  less  with  Speech:  Operators  more 
frequently  completed  trials  within  the  experimenter- 
specified  time  limit  with  Speech  Input. 


V  J  Results  -  Speech  Better  jT)? 

Subjective  Data  V 

i 

•  Manual  “more  difficult”  than  Speech 

OPERATOR  .  Manual  “higher  workload”  than  Speech 
RATINGS 

•  Manual  worse  than  Speech  for: 

•Interference  with  flight/navigation  task 
•Speed  and  accuracy  of  data  entry 

MANUAL:  “Menu... unintuitive.”  “Info 
buried  too  deep.”  “Have  to  be  heads 
down.” 

OPERATOR  voiCE:  “Voice  was  iust  so  much  easier.” 
COMMENTS  “You  can  stay  heads-up.” 

“Greatly  reduced  workload.” 

Operator’s  Console  Example 
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Operator  Vehicle  Interface 
Workstations 
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Mission  Control  Station 
Experimental  Laboratory 


Operator  Console  Development 


Partial  Immersion 

3-D  Multi-Sensory  Displays 
Synthetic  Vision 
Tactile  Displays 
|  Speech  Control 
^S^atial  Audio 

I  Supervisory  Control  Interface 

"UAVtUCAV  Application - 

50%  Virtual  Technology 
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No  Virtual  Technology 


Multi-Sensory  Interface 

Technology  Progression 


1999 


2003 


2005 


w  Operator-Vehicle  System  Diagram 

_ 


Devices 


[Operator State  Estimator  [ 


-Operator  State- 


OVI 

Adaptation 

Algorithm 

JL 


Situatlo 


Situation 


29 


ON 


-s; 


AIRBUS  SEEKS  TO  KEEP  PILOT,  J 
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Human  and  Machine  Co-operate 
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Increasing  Automation 
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Associate  System 


*  Associate  systems  are  computer-based  aiding  systems  that 
are  intended  to  operate  as  an  associate  to  the  human  user. 
(Geddes,  1997,  p.221) 

*  A  member  without  full  status  or  privileges. 

(Webster's  New  World  dictionary  1968,  p.  89) 

Mixed  Initiative  --  both  human  and  decision  aid  can  take 
action. 

Bounded  Discretion  -  the  human  is  in  charge. 

Domain  Competency--  decision  aid  has  broad  competency, 
but  may  have  less  expertise  than  its  human  counterpart. 
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Adaptive  Automation 


In  adaptive  automation,  the  level  or  mode  of 
automation  or  the  number  of  systems  that 
are  automated  can  be  modified  in  real-time. 
Furthermore,  both  the  human  and  the  machine 
share  control  over  changes  and  the  state  of 
automation. 


Scerbo,  p.  43  in  Atfomaiton  and  Human  Performance,  1906. 
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Human-machine  Control  Architectures 
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(a)  Manual  Control 

>  Goals 
'  Function 
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(c)  Co-operative  Functionings 


(b)  Supervisory  Control 
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Key:  H=  Human  M=  Machine 
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Level  of  Automation  and 
Human/System  Roles 


ra, 


Level  of  Automation 

Human  Role 

System  Role 

1.  None 

Decide,  Act 

2.  Decision  Support 

Decide,  Act 

Suggest 

3.  Consensual  Al 

Concur 

Decide,  Act 

4.  Monitored  Al 

Veto 

Decide,  Act 

5.  Full  Automation 

— 

Decide,  Act 

Endsiey,  Automation  and  Siturtion  Awareness,  p.  174,  1996. 


Three  Phases  of  Trust  Building 


*  Early  Phase:  Based  upon  the  predictability  of 
another's  behaviors 

*  Intermediate  Phase:  Dependability  --  "Summary 
statistic  of  an  accumulation  of  behavioral  evidence, 
which  expresses  the  extent  to  which  another  person 

may  be  relied  upon"  {Mur.  Intacnatbnai  Joomei  of  Man-Macline  Studies,  1987,  532.) 

*  Final  Stage:  Faith  that  a  person  will  be  dependable 
in  the  future. 
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Pilot's  Associate  Program 

Program  Concept 
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J  Prime  Directives  for  Associate  Systems: 
Example-Three  Laws  of  Robotics 


•  A  robot  may  not  harm  a  human  being,  or,  through 
in  action,  allow  a  human  being  to  come  to  harm. 

•  A  robot  must  obey  the  orders  given  to  it  by  a 
human  being  except  where  such  orders  would 
conflict  with  the  first  law. 

•  A  robot  must  protect  its  own  existence  as  long  as 
such  protection  does  not  conflict  with  the  first  or 
second  law. 


Ajmtuv.I.  t. 


Robot.  I960 
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4  A 

The  Effort  Required  of  the  Pilot  to  Control  the  PA 
Must  Be  Less  Than  the  Effort  Saved  by  the  PA. 


Plans  may  be: 

Approved  or  rejected  explicitly 

Approved  or  rejected  pre-mission 

Approved  or  rejected  implicitly  by  pilot  action 

Ignored,  with  predictable  results 

The  PA  must  operate  in  a  predictable  manner. 

All  PA  actions  must  be  tailorable  by  the  pilot. 

The  PA  is  required  to  monitor  the  pilot,  not  the 
other  way  around. 

The  PA  must  follow  the  pilot’s  lead. 
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'-I  Tiuz&r'jruji  Plluite  Associate 


Focused  Digital  Battlefield  Information 

•  Process  Intel  "real-time”  from  ell  available  resources 
■  Gather  Intel  data  to  relay  to  higher  echelons 


Real-time  Situational  Assessment 

•  Reasons  on  info  to  assess  battlefield  situation 
and  aircraft/team  members'  conditions 

•  Infer  the  enemy's  intent 

Cockpit  Information  Management 

•  Provides  pilot  selectable  task  automation 

•  Monitor  crew  actions/reactions  to  predict 
information  required  for  situation 

•  Present  info  to  crew,  when 
needed,  in  Form  most 
easily  understood 


Real-time  Computer-Aided 
Decision  Making 

Understands  commander's  intent;  infers  pilot’s 
Intent  4  plans  accordingly 

Presents  options  for  changing  battlefield  situations 
&:  pilot  makes  dicisions 

Plans  for  Route,  Rocon,  Attack,  Com,  Survivability, 
and  Sensors 


Sourc**  artTvy.fr. aDOul'f p»Ov00 .Niti  4 
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Longbow  Apache 


^  ^  Rotorcraft  Pilot’s  Associate:  Real-time 
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Computer-aided  Decision  Making 


•  Understands  Commander's  Intent; 
Infers  Pilots  Intent  and  Plans 
Accordingly 

•  Presents  Options  for  Changing  Battlefield 
Situations,  and  Pilot  Makes  Decisions 

•  Plans  for  Route,  Recon,  Attack,  Com, 
Survivability,  and  Sensors 
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Increased  Workload 


rVN° 


■  t  AJW~r  ’Tf* 

ij >•*  v-  ij  'v,'! 


3  - 

s 

a 

o 

0) 

g; 

f*T 

Adaptiveness 

“I  have  to  watch  Robonaut  closely  to 
make  sure  the  job  is  done  properly.” 


Workload,  Unpredictability  and 
Adaptiveness 


Increased  human  Increased  automation 

management  role  yields  management  role  yields 

increased  human  workload  increased  unpredictability 

to  human 


Increased  Unpredictability 
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w  Plan  Generated  From  GUI  Playbook 
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Finish  PUn  Final  Fkxito  Print  ScJiwJuia  Show  Sctwdute  :  Abort  i  Ou.i 
Figure  9.  Finished  plan  returned  by  the  MAC  after  User  specification  of  the  Ingress  task 
Miller,  Pelican  and  Goldman,  1999 
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Current  Research  Program 
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Affordability  will  result  in  a  high  degree  of  automation  in 
the  UMV  operator’s  console. 

Control  of  the  UMV  will  be  defined  more  in  terms  of  mission 
management  than  manual  control. 

Changes  on  the  battlefield  will  require  a  very  flexible 
automation  architecture. 

Successful  team  building  between  the  operator  and  the 
automation  will  require  trust  between  the  two. 

The  UMV  operator/console  system  is  a  key  feature  in 
achieving  future  mission  success. 


Slide  1  Contains  four  pictures:  A)  Across  the  top  of  the  slide  are  the 
Summary  Panels,  one  for  each  of  the  UAVs.  The  summary  panels  give 
the  operator  a  quick  view  of  the  status  of  each  of  the  UAVs.  B)  shows  the 
command  window.  Two  very  important  features  are  the  yellow  and  red 
boxes.  They  enable  the  operator  to  disconnect  from  the  boom  after 
refueling  and  breakaway  if  an  emergency  occurs.  C)  The  Fuel  System 
display  shows  the  total  quantity  and  individual  fuel  tank  quantities  for  the 
selected  UAV  ,D)  The  refueling  status  display  contains  three  lights.  The 
lights  are  blue  for  ready  (RDY),  green  to  show  contact  with  the  boom  (AR), 
and  yellow  to  show  the  boom  disconnect  (DISC).  The  total  fuel  quantity  of 
each  UAV  is  shown  below  these  three  lights. 

Slide  2  shows  The  Out-the-Window  display  which  was  a  virtual  display 
depicting  the  positional  information  of  the  UAVs  and  tanker. 

Slide  3  shows  position  information  about  the  UAVs,  tanker,  commanded 
positions.  It  is  the  primary  mechanism  for  commanding  the  UAVs  during 
the  air  refueling  process. 

Slide  4  shows  that  once  the  UAVs  were  positioned  at  the  observation 
positions,  and  the  UAV  operator  received  permission  from  the  boom 
operator  to  move,  the  operator  used  the  mouse  to  drag  the  selected  first 
UAV,  Striker  01 ,  to  the  precontact  position. 

Slide  5  shows  that  the  Striker  01  is  now  in  the  contact  position. 

Slide  6  shows  that  all  of  the  Strikers  have  been  refueled  are  now  in  the 
post  refueling  position. 


Slide  1  NOTE:  Slides  printed  on  both  sides 
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